Negative Learning Rates and P-Learning

We present a method of training a differentiable function approximator for a regression task using negative examples. We effect this training using negative learning rates. We also show how this method can be used to perform direct policy learning in a reinforcement learning setting. 1 Regression and Learning Rates The goal of regression analyses is to find a regression function, a function that models the relationship between the independent variables (the inputs) and the dependent variables (the outputs). When a complex relationship between these variables, an exact regression function is not sought. Instead an approximate regression function is used to model the relationship. Function approximators have been shown to be a sound method for finding approximate regression functions. Function approximates are generally trained using example input-output pairs from the function that is to be approximated. Given the input from the pair, the output of the function approximator is compared to the actual output from the example. The function approximator is then modified –possibly through gradient descent– so that its output matches the output in the example. This is one training step. The function approximator is not usually modified so vigorously that the output matches the example output perfectly for each training. Extreme modification of the approximator tends to result in the loss or forgetting of the previous examples, that is, the previous training is overwritten by too strong of an update. Also, updates that are too vigorous tend to negatively affect the approximator’s ability to generalize to unseen examples. Another issue occures when a function approximator is updated, the parameters of the model, it are changed. If these parameters are changed too quickly, they can overrun a computer’s ability to represent these numbers. This is sometimes known as model explosion and it prevents the use of the function approximator. To solve these and other problems, the updates to a function approximator are attenuated by a fractional amount, conceptualized as the learning rate. This allows the approximator to be pushed in the desired direction by a small, tunable amount. All approximators trained by gradient descent use a learning rate. Numerous methods have been developed to automatically compute learning rates. Many use the second derivative of the parameter change (where the first derivative is the raw magnitude and direction of the update. This basic method is used from the familiar Newton’s Method, to esoteric concave optimization methods. Other methods use previous update amounts to tune the learning rate. However, the most common method for setting a learning rate is to start with 0.1 and hope for the best. 1 ar X iv :1 60 3. 08 25 3v 1 [ cs .A I] 2 7 M ar 2 01 6 2 Learning Rates as a Learning Channel In regression, the learning rate does not contain information about what the approximator should output for any given input. This information is contained in the input-output example pairs. This is why the learning rate is always positive; a positive learning rate means that the approximator should match this example more closely. Still, there are many situations where we do not have access to good input-output pairs. The pairs we have access to might not be from the actual function we are trying to model, but from any other function. The pairs we have might be totally random. However, we might have access to a measure of how closely the output we have matches up with hypothetical output, given some input, from the function we are trying to approximate. For example, take input-output pair e = (x, r) where x is some input vector and r is a random output vector. We would like our function approximator nn(x) to give us the correct output from our target function nn∗(x) = y, but we do not have access to any example nn∗(x) = y for any x. However, if we have a distance function dist∗(x, r) → R that gives a similarity measure between r and nn∗(x) = y, we can still train nn(·) We would like to use dist∗(·, ·) as a training signal to train n. Learning rate is useful as a channel for this distance training signal. To use the learning rate as a learning channel we can take each example in the training set and use it to assign a custom learning rate for each example. We should still have a global learning rate μ. Now, when we train on example i, the learning rate for example ei is μ× dist(xi, ri). Now we train a simple 2-layer artificial neural network to reproduce the sine function on the interval [−5, 5] with 40 example points. The network uses tanh activation function for the 128 hidden units. Each example is of the form ei = (xi, ri, di) where xi = −5 + i × 0.25 and ri is drawn from a uniform distribution on the interval [−1, 1]. The value di is the examples learning rate factor calculated by dist(xi) = di = |sin(xi)− ri|. Let’s see how this works in figure 1. Figure 1: Output of network trained with per example learning rates di = dist(xi, ri).